fix(python): make dataframe index opt-in - #919
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #919 +/- ##
========================================
Coverage 61.26% 61.26%
========================================
Files 746 746
Lines 51560 51560
Branches 8353 8353
========================================
Hits 31586 31586
Misses 18359 18359
Partials 1615 1615 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR updates the Python TsFileDataFrame to make the persistent mmap-backed dataset index opt-in (use_index=True), preserving the legacy direct metadata scan by default and preventing unintended .tsidx creation/reuse. It also scopes the persistent index filename by hashing the expanded canonical file set so different subsets under the same directory don’t contend for the same index file.
Changes:
- Add
use_index: bool = FalsetoTsFileDataFrameand route metadata loading through legacy scan when indexing is disabled. - Make persistent index filenames unique per canonical TsFile set via a SHA-256 digest in
index_path_for(). - Update tests to cover default no-index behavior, index isolation for different subsets, and explicitly enable indexing where runtime/index behavior is required.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| python/tsfile/dataset/index.py | Hash-scopes the .tsidx filename to the canonical TsFile set. |
| python/tsfile/dataset/dataframe.py | Adds use_index flag and supports legacy querying paths when runtime/index is disabled. |
| python/tests/test_tsfile_dataset.py | Parameterizes many dataset tests to run with and without persistent indexing; forces runtime-specific tests to use indexing. |
| python/tests/test_dataset_index.py | Adds explicit coverage that default construction doesn’t create an index and that different file subsets create/reuse isolated indexes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| def index_path_for(paths: Sequence[str]) -> str: | ||
| common = os.path.commonpath([os.path.abspath(path) for path in paths]) | ||
| canonical_paths = sorted(os.path.abspath(path) for path in paths) | ||
| common = os.path.commonpath(canonical_paths) |
Description
This PR makes the Python
TsFileDataFramepersistent dataset index opt-in.By default,
TsFileDataFrame(..., use_index=False)now keeps the previous direct metadata loading behavior and does not create or reuse any.tsidxfile. Users can explicitly enable persistent indexing withuse_index=True; when enabled, the dataframe will reuse a matching index if present, or build one when the matching index is missing or stale.The persistent index file name is now scoped by the expanded canonical TsFile set, so different subsets under the same directory no longer contend for the same
.tsidxfile.Changes
use_index: bool = FalsetoTsFileDataFrame.use_index=True.Validation
./mvnw -P with-cpp -pl cpp -DskipTests package./mvnw -P with-python -pl python test200 passed